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In a previous paper (paper I), we derived a set of near-optimal signal detection techniques for 
gravitational wave detectors whose noise probability distributions contain non-Gaussian tails. The 
methods modify standard methods by truncating or clipping sample values which lie in those non- 
Gaussian tails. The methods were derived, in the frequentist framework, by minimizing false alarm 
probabilities at fixed false detection probability in the limit of weak signals. For stochastic signals, 
the resulting statistic consisted of a sum of an auto-correlation term and a cross-correlation term; 
it was necessary to discard "by hand" the auto-correlation term in order to arrive at the correct, 
generalized cross-correlation statistic. 

In the present paper, we present an alternative derivation of the same signal detection techniques 
from within the Bayesian framework. We compute, for both deterministic and stochastic signals, the 
probability that a signal is present in the data, in the limit where the signal-to-noise ratio squared 
per frequency bin is small, where the signal is nevertheless strong enough to be detected (integrated 
signal-to-noise ratio large compared to one), and where the total probability in the non-Gaussian 
tail part of the noise distribution is small. We show that, for each model considered, the resulting 
probability is to a good approximation a monotonic function of the detection statistic derived in 
paper I. Moreover, for stochastic signals, the new Bayesian derivation automatically eliminates the 
problematic auto-correlation term. 



I. INTRODUCTION AND SUMMARY 

Most of the literature on gravitational-wave data anal- 
ysis assumes that the detector noise is Gaussian. How- 
ever, significant non-Gaussian tails have been a char- 
acteristic feature of the noise distributions in all gravi- 
tational wave detectors constructed to date. Standard 
detection strategies for both deterministic and stochas- 
tic signals, which were designed under the assumption of 
Gaussian noise, perform more poorly when non-Gaussian 
noise is present. 

In a previous paper in this series jjj (henceforth paper 
I) , we developed a new set of statistical signal-processing 
techniques to search for deterministic and stochastic 
gravitational waves in detector data. These techniques 
are robust, meaning that they will work well even if the 
detector noise is not Gaussian but falls into a broader sta- 
tistical class that we expect includes realistic detectors. 
These new methods are similar to the older ones: one 
constructs matched filters to search for known waveforms 
or cross-correlates the instrument outputs at the differ- 
ent detector sites to search for a stochastic background. 
The essential difference is that the statistics are modi- 
fied by truncation: detector samples that fall outside the 
central Gaussian-like part of the sample distribution are 
excluded from (or saturated when constructing) the mea- 
surement statistic. For both deterministic and stochas- 
tic signals, a robust statistic was found which performs 
better than the optimal linear filter in the case where 



the detector noise is non-Gaussian, and almost as well in 
the Gaussian-noise case. Alternative methods for dealing 
with non-Gaussian noise for stochastic signals have been 
explored by Klimenko and Mitselmakher [^) . 

In paper I, we derived the statistics using the frequen- 
tist criterion of minimizing false alarm probabilities at 
fixed false detection probabilities in the limit of weak 
signals. In the present paper, we present an alternative 
derivation of the same signal detection techniques from 
within the Bayesian framework. 

We start in Sec. II by reviewing the foundations of the 
two different approaches to determining detection statis- 
tics used in paper I and this paper. We review the locally 



optimal criterion used in paper I in Sec. II A. In Sec. II B 



we explain how Bayesian considerations lead to a unique 
choice of detection statistic, as discussed by Finn ||. In 
Sees. [II and |y| of the paper, we compute, for a variety 
of different models and sets of assumptions, that unique 
Bayesian statistic. We show that for each case consid- 
ered, the Bayesian statistic is equivalent to the statistics 
derived in paper I, in the sense that the false alarm ver- 
sus false dismissal curves of the two statistics coincide to 
a good approximation. 

The equivalence between the two types of statistic is 
valid only under certain approximations, discussed be- 
low. Under those approximations, the Bayesian statis- 
tics which we obtain arc equivalent to a particular type 
of maximum likelihood statistic described in Refs. ||, [|. 
That type of maximum likelihood statistic differs from 
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the type of maximum likelihood statistic considered pre- 
viously in gravitational wave data analysis in its treat- 
ment of noise parameters. 

Section III deals with known, deterministic signals. 
In Sec. [II A we consider the case of a known signal of 
unknown amplitude, incident on a single detector with 
white, Gaussian noise, where the noise variance is as- 
sumed to be known. For that case the Bayesian statistic 
is shown to be equival ent to the standard matched fil- 
tering statistic. Section III B generalizes this analysis by 
allowing the noise variance to be an unknown parame- 
ter, to be measured from the data; the same result is 



obtained. In Sees. [II C and III D we consider the cases 
of white and colored non-Gaussian detector noise. For 
both of these cases we show that the Bayesian statistic is 
equivalent to the corresponding locally optimal statistic 
derived in paper I. 

In Sec. IV we gi ve a similar analysis of stochastic sig- 
nals. In Sec. [V A we compute the Bayesian statistic for 
the case of a white stochastic signal, and of two co-aligned 
detectors with white Gaussian noise, where the noise vari- 
ance is assumed to be known. The maximum likelihood 
statistic for this case was previously computed, in a more 
general context, by Finn and Romano In this case 
we recover the result of Finn and Romano: the opti- 
mal statistic is not the standard cross-correlation statis- 
tic, but instead is a sum of the cross-correlation statis- 
tic and and extra auto-correlation terms. In Sec. IV B 



we show that, under the more realistic assumption where 
the noise variances are taken to be unknowns to be deter- 
mined from the data, then the standar d cros s-correlation 
statistic is recovered. Finally, in Sec. IV C we consider 
two detectors with white, non-Gaussian nosie, and we re- 
derive the generalized cross-correlation statistic of paper 
I. Section |v| contains a short conclusion and summary. 



II. FOUNDATIONS 

We denote the output of a set of gravitational wave 
detectors by a vector x, with 

x = n + s. (2.1) 

Here n is the detector noise, and s is a possibly present 
gravitational wave signal. We can write 



es, 



(2.2) 



where e is a parameter governing the signal strength, and 
the magnitude of s is fixed. As in paper I, we shall be 
specializing to weak signals and using an expansion in 
powers of e about e = throughout this paper. 



A. Frequentist signal detection 

A key quantity is the probability distribution for x 
given e, p(x|e). This quantity can be used to compute 



the performance of detection statistics in the frequentist 
framework. Suppose one is given a statistic 



r = r(x), 



(2.3) 



say, and that one's detection criterion is that a signal is 
present if T(x) > and not present otherwise, where 
r» is a threshold. Then the false dismissal probability 
associated with this statistic is 



a(r.) 



dxp(x|Q), 



T>r» 

and the false alarm probability is 



/?(r*,e) 



dxp(x|e). 



(2.4) 



(2.5) 



r<r, 



By eliminating r» between Eqs. (2.4) and (2.5) we obtain 
the false-dismissal versus false-alarm curve 



/? = /?(<*, e) 



(2.6) 



which characterizes the performance of the statistic. 

In paper I we showed that there is a unique statistic 
A(!)(x) which minimizes dp/de at e = for fixed a, de- 
fined by the expansion 

p(x|e) = p(x|0) [1 + eA (1) (x) + e 2 A (2) (x) + 0(e 3 )] . 

(2.7) 

This statistic therefore has the best false-dismissal versus 
false-alarm curve for weak signals. [If A(i)(x) vanishes 
identically, then A( 2 ) (x) is the unique statistic that min- 
imizes d 2 (3/de 2 at e = for fixed a.] We applied this 
class of detection statistics (called locally-optimal statis- 
tics H ) to a variety of different gravitational- wave signal 
detection problems. 



B. Bayesian signal detection 

In the Bayesian framework, the probability pW that 
a signal is present in the data is given 



p(i) 
1 - P« 



A(x) 



p(0) 

1 _ p(o) ■ 



(2. 



where P*- -* is the a priori probability that a signal is 
present, and A(x) is the likelihood ratio. In the litera- 
ture on Bayesian statistics A(x) is called a Bayes factor. 
The Bayesian framework uniquely determines a detection 
criterion, which is to threshold on t he probability P^ 1 ' 
that a signal is present. From Eq. (2.8) it is clear that 
PW is a monotonic function of A(x), so one can equiv- 
alently threshold on A(x). Thus, an optimal detection 
statistic is uniquely determined in the Bayesian frame- 
work; however, that statistic does depend on choices of 
prior probability distributions. 

We now describe how likelihood ratio is computed. It 
is given by the formula 



A(x) = / <feA(x,e)p (0) (e) 



(2.9) 
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where p^ (e) is the prior probability distribution for the 
signal strength e, and 



A(x,e) = 



P( x k) 
p(x|0)' 



(2.10) 



Suppose that the noise n is described by a probability 
distribution p n (ja), and the signal s by a s ignal distribu- 
tion p s (s|e). Then it follows from Eq. (2.1) that 



p(x|e) = J dsp n (x - s)p s (s|e). (2.11) 



The formula (2.10) can therefore be written as 



A(x,e)= / ds Pn{ * . S) p s (s|e). (2.12) 
J Pn(n) 

The formula for A(x, e) becomes more complex when 
there are unknown signal and/or noise parameters 
present. Suppose the signal distribution depends on 
some parameters 8 S in addition to the signal amplitude 
e, which themselves are distributed according to a prior 
probability distribution pg g (8 s \e). Then the signal distri- 
bution p s (s|e) can be expanded as 



p s (s|e)= f de s p 8 {s\e,e s )pe t (e s \e), 



(2-13) 



where f> s (s|e, 8 S ) is the distribution for s given both e and 
8 S . Similarly suppose that the noise distribution contains 
unknown parameters 8 n , whose a priori distribution is 
pg n (O n ). Then the noise distribution can be expanded as 



p n (n) = J dd n p n (n\8 n )pg n (8 n ). 



(2.14) 



Inserting the expansions ( |2~1~3| ) and ( |2~l"4l ) into Eq. ( pl^ ) 
gives the final expression for the likelihood function: 

/ dd n p a (x.-s\e n )pe n (On) 



A(x,e) = ds 



fd9' nPa (x\8' n )p e M 
dO sPs (s\e,8 s )po s (8 8 \e). (2.15) 



Equations (23) and ( 2.15| ) are the foundational equa- 
tions that we will use throughout this paper to compute 
the likelihood ratio A(x). A key feature of this formalism 
is that the noise parameters 8 n are treated as unknowns, 
to be measured from the detector data along with the 
gravitational wave signal, rather than being treated as 
known a priori. That feature underlies the elimination of 
the auto-correlation terms encountered in paper I in the 
case of a stochastic gravitational wave background. 

In the next few sections we will revisit several of the 
signal detection problems considered in paper I. In each 
case, we will show that A(x) is, to a good approxima- 
tion, a monotonic function of the locally-optimal detec- 
tion statistic derived in paper I. Since a monotonic func- 
tion of a detection statistic has the same false alarm ver- 
sus false dismissal curve as the original statistic, it fol- 
lows that in each case the uniquely determined Bayesian 



statistic A(x) is equivalent to the statistic computed in 
paper I. Note, however, that the equivalence only applies 
at the level of choosing the detection statistic, and not at 
the level of specifying thresholds. The Bayesian and fre- 
quentist approaches lead to different detection thresholds 
for a given specified significance level; see, for example, 
the discussion in Sec. III.C of Ref. [Q. 

In deriving the formulae for the likelihood ratio A(x), 
we shall invoke a number of different approximations. In 
assessing the validity of those approximations, we shall be 
concerned only with their effect on the false alarm versus 
false dismissal curve of the statistic. In other words, the 
approximations might be very inaccurate for computing 
the value of A(x), but might nevertheless be very accu- 
rate in the sense that they have only a small effect on the 
false alarm versus false dismissal curve. [We do need to 
compute accurate numerical values of A(x), since we are 
not concerned here with computing detection thresholds.] 
We shall use the notation 



A 1 (x)~A 2 (x) 



(2.16) 



to mean that the false alarm versus false dismissal curves 
of the statistics Ai(x) and A 2 (x) are approximately the 
same. 



III. DETERMINISTIC SIGNALS 

A. Single detector, white Gaussian noise, known 
variance 

We first treat the simple case where we are looking for 
a signal s, in a single detector, whose values in the time 
domain are [Q 

Sj=es 3 . (3.1) 

We assume that the quantities Sj are known and fixed, so 
that the only unknown parameter characterizing the sig- 
nal is its amplitude e, which can be positive or negative. 
Without loss of generality we can choose the normaliza- 
tion so that 



= i. 



(3.2) 



We assume that the detector noise is white and Gaussian 
with zero mean and unit variance. Then, as shown in 
paper I, the distribution for the data x given e is 



2 J 



Inserting this formula into Eq. ( 2.1 0| ) gives 
A(x,e) = exp [ee(x) -e 2 /2] , 

where 

e(x) " 
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(3.3) 



(3.4) 



(3.5) 
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is the standar d m atched filtering statistic. Combining 
this with Eq. (2J3) gives for the likelihood ratio 



A(x) 



e(x) 2 /2 



dep {0 \e)e 



«(x)] 2 /2 



(3.6) 



Now the quantity |e(x)| is effectively the signal-to- noise 
ratio. Let us assume that we are in the relevant regime 
where the signal is detectable with high confidence, so 
that 



exp [e(x) 2 /2] > 1. 



(3.7) 



Let us also assume that the prior distribution p^ (e) is 
slowly varying and does not strongly constrain the possi- 
ble values of e. Then, we can approximately evaluate the 



integral (3.6) using the Laplace approximation to obtain 



A(x) « vW } [e(x)] exp [e(x) 2 /2] 



(3.8) 



Finally, we argue that we can neglect the depe ndence on 
x of the factor p^ [e(x)] in the expression (3.8). The rea- 
son is that the prior distribution p(°> (e) is a slowly vary- 
ing function of e, and so this factor has a much weaker de- 
pendence on x than the exponential factor in the regime 
(3.7). Therefore, dropping the factor p(°)[e(x)] will have 
a negligible effect on the false alarm versus false dismissal 
curve of the statistic. In this approximation we see that 
A(x) is a monotonic function of the standard detection 
statistic |e(x)|, 



A(x) ~ exp [e(x) 2 /2] , 



(3.9) 



as claimed |l3| . 

We remark that there is a key technical difference 
between the above computation and the corresponding 
computation in Sec. II. A. of paper I. The Bayesian com- 
putation presented here requires expanding the quan- 
tity lnA(x, e) to second order in e about e = [Eq. 
fl3.4p above], whereas in paper I it sufficed to compute 
In A(x, e) to linear order in e [Eqs. (2.3) and (2.5) of pa- 
per I] . This difference is a common feature of all of our 
subsequent computations. 



B. Single detector, white Gaussian noise, unknown 
variance 

We now add one additional complication to the analy- 
sis, by taking the noise variance to be an unknown con- 
stant a. We define an inner product ( , ) on the space of 
signals by 



E 



(3.10) 



Note that this is not the standard inner product used 
in discussions of matched filtering, which incorporates a 
weighting factor of er -2 . We define the statistic <r(x) by 



a(x) 2 



1 

TV 



(3.11) 



where N is the number of data points; this is the con- 
ventional estimator of a. We also define the quantity p 
by 

p=-, (3.12) 



which from the normalization condition (3.2) is the con- 



ventional signal to noise ratio. The corresponding esti- 
mator is 



/5(x) = 



The conventional matched filtering statistic is 

- (x, s) , 
a 



(3.13) 



(3.14) 



and if we replace the noise variance by its estimator <r(x) 
we obtain the statistic 



Pi( x ) 



1 



«r(x) 



(x, s 



(3.15) 



We shall show below that the likelihood ratio A(x) is 
to a good app roximation equivalent to the conventional 
statistic ( 3.15 ). 

The noise distribution given a is taken to be 



Pn(n\o-) = TT — =^exp 

V27T<7 



3 

2a 2 



(3.16) 



The full noise distribution is [cf. Eq. (2.14) above] 

p a (n) = / dap a (a)p a (n\a), (3.17) 



where p a {o~) i s the prior pro babili ty distribution for a. 
Inser ting Eq. fl3.17| ) into Eq. ( |3.16D and using the defini- 
tion ( 3.11 ) we obtain 



/>OC 

Pn(n) - / da Pa (a) exp [-NE(a)/2] , 
Jo 



where 



E(a) = ln(27rcr 2 ) + 



(3.18) 



(3.19) 



We can approximately evaluate the integral ( 3.18| ) in the 
limit where N is large. The function S(er) can be ex- 
panded about its local minimum at u — <r as 

E(<t) = l + H2^& 2 ) + ^{a-a) 2 +0[{a-af]. (3.20) 
o~ 

Using this expansion we obtain 



p n (n) = t /-(2,e)-^[a(n)]a(n)-( w - 1 ) 



1 + (-= . (3.21) 
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We assume that P a(cr) is slowly varying, and so have 
neglected in Eq. ( 3.21 ) a fractional error of order the 
fractional change in p a over a i nterv al of width a/\f~N . 

We next insert the form ula ( 3.21) for the noise distri- 
bution into the expression (2.12) for the likelihood ratio, 
using 



p s (s|e) = ^(s - es) 



The result is 



P<t[ct(x - es)] 



A(x)= / depW(e)-. 

Jo ?Mcr(x) 

P<t[ct(x - es)] 



it(x — es) 



a(x) 



<fep (0) (e) : - , 



x exp 



(3.22) 



n -(JV-l) 

(3.23) 



(3.24) 



To obtain the second line we used Eqs. (3.2) and (3.11). 
Expanding the logarithm to second order in e, we can 
re-express this as 



A(x) 



dep{0) p^x-es)] 

' PoM X )J 



x exp 



ae - be 2 + 0(e 3 



where 



and 



(N-l) 



(x, a) 
(x,x) 



6=-(JV-l) 



1 2 (x,s) 



^x,x) 



(3.25) 



(3.26) 



(3.27) 



Before proceeding further with the computation of the 
likelihood ratio, we clarify the domain of validity of the 
weak signal expansion (expansion in powers of e) used in 
going from Eq. ( |3.24j ) to Eq. ( |3.25| ). We will estimate the 
expected sizes and the scale of statistical fluctuations in 
the two term s appearing in the argument of the logarithm 
in Eq. ( 3.24 ); the expansion will be good when both of 
these terms and their fluctuations are small compared to 
unity. For the purpose of making these estimates we can 
identify a and a, and p, p, and p\. 

We can compute the expe cted v alue and v arian ce of the 
statistic (x, x) using Eqs. ( |2.l| ), (2.2) and ( 3.16 ), which 
gives 



{Na 2 + e 2 )± y / 2Na 4 + 4e 2 cr 2 . 



(3.28) 



The notation here is that the first term gives the ex- 
pected value, and the second term gives an estimate of 
the statistical fluctuations. We can rewrite this formula 
in terms of the signal-to- noise ratio p — e/a as 



(x,x) 



Na 1 



(l + ^)± 



2 

N 



1^1 

N 2 



(3.29) 



Similarly we have 



x,s 



e±cr. 



(3.30) 



We assume that N ^> 1 and that p > 1. We now consider 
two different cases: 

• When p 2 /N <C 1, the fluctuations in (x, x) are 
small compared to the expected value, and we have 
from Eq. (|3.29[) that (x,x) ~ Na 2 ± y/Na 2 . Us- 



ing this together with Eq. (3.30) shows that the 
first t erm in the argument of the logarithm in Eq. 
(HI is 



(x, s 



\x,x 

and similarly the second term is 

,2 



(x,x) 



P 2 , P 2 

N N 3 / 2 



< 1. 



(3.31) 



(3.32) 



Thus the approximation is good in this regime. 

• When p 2 /N 3> 1, a similar computation gives that 
(x, x) ~ p 2 a 2 ± pa 2 . The the first term in the 



argument of the logarithm in Eq. ( 3.24 ) now scales 
as 



1 

1 ± - 

P 



and similarly the second term scales as 
- 2 1 



1 ± -. 

P 



(3.33) 



(3.34) 



Thus, the approximation breaks down in this 
regime. 

We now return to computing the likelihood ratio A(x). 
We can approximately evaluate the integral ( 3.25| ) using 
the Laplace approximation to obtain 



77 . (3-35) 




e(x) = 



«( x ) 
26(x) ' 



(3.36) 



This approximatio n wil l be good whenever the exponen- 
tial factor in Eq. ( 3.35|) is large, which it will be in the 
regime where the signal is detectable (see below), and 
when the prior probability distribution pw(e) is slowly 
varying. Using Eqs. ( pllf ), ( pT5[ ), jO^ and fiTTfy , we 
can write the exponential factor as 



exp 



N-l 



9{pi/N) 



(3.37) 
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where the function g is given by g(x) = x/(l — 2x). Since 
we are in the regime p 2 /N <C 1, the argument of the 
function g is small, and we can replace g{p\/N) by p\/N. 
This gives, using N 3> 1, 



A(x)«p(°)(e) 



P<r[0-(X - 




exp 



1 



;Pi(x) 2 



p CT [a(x)j y b 

(3.38) 

we argue as before that in the regime 
^> 1 where the signal is detectabl e, the de- 
pendence on x of all the other factors in Eq. (3.38) can 
be neglected in comparison to the exponential factor, as- 
suming that the prior distributions are slowly varying. 
This gives 



Finally, 
exp[p 2 /2] 



A(x) ~exp[pi(x) 2 /2] 



(3.39) 



as claimed. 

Our fi nal answer ( 3.39 ) is essentially the same as the 
answer ( |3.9[ ) obtained when the noise variance a is as- 
sumed to be known. Therefore, treating a as an unknown 
parameter rather than as a fixed, known parameter does 
not make much difference in this case. However, we will 
see below for the case of stochastic signals that treating 
the properties of the noise distribution as unknowns does 
have a significant effect on the analysis, and that the cor- 
rect answer is obtained only when the those properties 
are treated as unknowns. 

We end this subsection by recapitulating the various 
approximations and assumptions we have invoked: 

• The large N approximation N 3> 1. 

• The assumption that we are in the regime where the 
signal is detectable, exp(p 2 /2) ^> 1. This is neces- 
s ary f or evaluating the integral over e to obtain Eq. 
(3.35), and also for the validity in neglecting the 
prcfactors in deriving Eq. ( 3.39] ). From a practi- 
cal point of view the assumption exp(p 2 /2) 3> 1 is 
not a serious restriction, as it does not matter how 
our statistics perform in the regime exp(p 2 /2) ~ 1 
where signals are not detectable. 

• The assumption that the prior probability distribu- 
tions j/°)(e) and p CT (cr) are slowly varying. 

• We have clarified the "weak signal" assumption of 
paper I; it is the assumption is that the signal-to- 
noise ratio squared per data point is small, p 2 /N <C 
1. This requirement ensures that the presence of 
the signal does not significantly bias the estimate 
(3.11) of the noise variance. In practice we can al- 
ways choose segments of data large enough to sat- 
isfy this assumption. 



C. Single detector, white non-Gaussian noise 

We now turn to the case where the noise has a known, 
non-Gaussian distribution. In this subsection we follow 



Sec. II. A. of paper I, and assume that the noise sam- 
ples in the time domain are statistically independent but 
identically distributed, with a known distribution. We 
can write the noise probability distribution as 



Pn(n) = Y[t 



-/("*) 



(3.40) 



We assume that the probability distribution e ^ ^ has a 
central Gaussian region |x| < Xb in which 



/(*) 



x 

2o*' 



(3.41) 



which contains most of the probability, and a tail region 

\x\ > Xb containing a total probability ptaii withptail <C 1- 
As before, the signal is assumed to be known up to 
an ov erall amplitude parameter. From Eqs. (2.11) and 
( 3.22 ) we obtain the following modified version of Eq. 
(pp 



p( x k) = n° xp \.~f( x 3 - e ^i)] ' 



(3.42) 



and inserting this into Eqs. (2.9) and (2.10) gives 
A(x) = J dep(°)( e ) []exp [-f( Xj - esj) + f( Xj )] . 



Expanding to second order in e gives 



A(x) 
where 

and 



e) exp 



o(x)e - 6(x)e 2 + 0(e 3 ) 



*(*) = £/'( 



X w J S 7 



&(x) = ££/"(*i)^ 



(3.43) 

)' . 
(3.44) 

(3.45) 
(3.46) 



Evaluating the integ ral ov er e using the same types of 
arguments as in Sec. IIIB gives 



A(x) ~ exp 



a(x) 2 

26(x) 



(3.47) 



Now the statistic a(x) is the locally optimal statistic 
computed in paper I [Eq. (2.9) of paper I]. Therefore it 
remains to show that we can neglect the x dependence 
of the factor fe(x) in the argument of the exponential in 
Eq. Q. 



We can split the sum ( 3.46 ) into contributions from 
the Gaussian region and from the tail. Using the fact 
that f"(x) = 1/er 2 in the Gaussian region, we obtain 



(3.48) 
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Most of the values of Xj will fall in the central Gaussian 
region, since Xj = rij + esj , and esj -C cr for each individ- 
ual j Q. Since J2j = 1, the first term gives l/(2cr 2 ), 
up to fractional corrections of order ptaii- Similarly the 
second term will be bounded above by ~ pmn/ (2a 2 ), 
since f"(x) will be smaller in the tails than in the central 
Gaussian region. We conclude that 



6(x) 



2a 2 



[1 + O(ptail)] • 



(3.49) 



It follows in particular that the x-dependent fluctuations 
in 6(x) are smaller than its expected value by a factor of 
Ptaii *C 1, and therefore we can neglect the x dependence 



of 6(x) in Eq. (3.47), as required 



As before, the statistic o(x) coincides with the locally 
optimal statistic derived in paper I [Eq. (2.21) of paper 
I], and it suffices to show that the x dependence of the 
factor 6(x) can be n eglec ted in Eq. (3.47). We evalu- 



ate the sums in Eq. (3.55) by splitting them into Gaus- 



sian and tail contributions as before. Since 



fltJ 



= 1 in 



the Gaussian region, the first term in Eq. ( 3.55 ) yields 
2 J2k \sk\ 2 / Pk[l + 0(ptaii)]- Also the second term is pro- 
portional to ptaii since g k (x) vanishes in the Gaussian 
region. Thus we obtain 



6(x) 



|S*| a 



1 + O(ptail) 



(3.56) 



and the rest of the argument follows as before. 



D. Single detector, colored non-Gaussian noise 

We next consider the model of colored, non-Gaussian 
noise of Sec. II. B. of paper I, where each frequency bin 
is assumed to be statistically independent. This is given 
by 



[(N-l)/2] 



Pn(n) 



n 

k=l 



exp 



-2g k 



where the volume element is understood to be 

[(JV-l)/2] 

mn k ). 



k=l 



Here 



-y 



e 2nijk/N 



(3.50) 



(3.51) 



(3.52) 



are the components of the discrete Fourier transform of 
the time domain samples rij . The quantities Pk describe 
the noise spectrum. For each frequency bin k, the func- 
tion gk{x) is arbitrary except for the normalization con- 
ditions 



dxe- 9k{x) = / dxxe~ 9k{x) = 1, 



(3.53) 



and the requirement that gk{x) — x in a central Gaussian 
region containing most of the probab ility. 

we again ar- 



By paralleling the a nalys is of Sec. QIC, 
rive at the formulae ( |3.44 ) and ( 3.47[ ), where now the 



statistics a(x) and 6(x) are given by 

fe | 2 \ Re(x* k s k ) 



Pk 



(3.54) 



and 



6(x) 



k 



\ikl_ 

Pk 



9k 



M 
Pk 

Pk 2 



(3.55) 



E. Signals with unknown parameters 

We now generalize the analysis of the preceding sub- 
sections by allowing the signals to depend on additional 
parameters other than the overall amplitude parameter 
e. We write 



es(0 s ), 



(3.57) 



where the signal parameters 6 S are distributed accor ding 
to the distribution pg s (O s \e). Then from Eq. (2.15) we 
can write 



A(x) = j de J d8 s p^(e) Pes (0 s \e)A(x,e,9 s ), 



(3.58) 



where A(x, e, 9 S ) is given by Eq. (2.11) with p s (s|e) re- 
placed by p s (s|e, S ). In the regime where the signal is 
detectable, we can repeat the arguments of the preceding 
subsections to approximately evaluate the integrals as 



A(x) ~ max max A(x, e, 6 S 



(3.59) 



Thus the result is to take the statistics previously de- 
rived and to maximize over the signal parameters. Such 
a maximization is the standard thing to do for linear 
matched filtering; the above argument indicates that it 
is also the appropriate procedure for the more general 
class of locally optimal statistics. 



IV. STOCHASTIC SIGNALS 

The standard method of detecting a stochastic back- 
ground is to compute a cross-correlation between two 
different instruments see Ref. [fill for a detailed de- 
scription. In Sec. 



IV A| below we compute the likelihood 



ratio A(x) for the simplest case of a white stochastic sig- 
nal, and of two co-aligned detectors with white Gaus- 
sian noise, where the noise variance is assumed to be 
known. For this case we do not recover the standard 



8 



cross-correlation statistic, but instead we obtain a statis- 
tic with extra auto-correlation terms. That statistic was 
first derived and has been investigated in detail in a more 
general context by Finn and Romano H . We then argue 
that it is unrealistic to take the noise variances to be 
known parameters. 



and where 



IV B we show that, when 



In Sec. 

the noise variances are taken to be unknowns to be de- 
termined from the data, then the likelihood ratio A(x) 
is to a good approximation equivalent to the standard 
cross-correlation statistic. This computation is again in 
the simple context of coincident alig ned d etectors with 
white noise. The computation of Sec. IV B] is a simplified 
version of the computation in Appendix A of Ref. . 

We then turn to non-Gaussian noise models. In pa- 
per I, we derived a generalized cross-correlation (GCC) 
statistic appropriate for non-Gaussian noise, which is a 
modification of the standard cross-correlation statistic. 



In Sec. IV C below we re-derive that statistic using the 



Bayesian approach. 



A. Two coincident co-aligned detectors, white 
Gaussian noise, known variances 

The output of the pair of detectors is x = (xi,X2), 
where 



xi = ni + s 
is the output of the first detector, and 
x 2 = n 2 + s 



(4.1) 



(4.2) 



is the output of the second. We assume, for simplicity, 
that the noise in each detector is white and Gaussian 
with unit variance: 



P ni (ni) = J|-=exp 



with a similar equation for the second detector. We as- 
sume that the stochastic background signal is also white 
and Gaussian with variance e: 



Ps(s|e) 




3 

2e 



(4.4) 



where e > 0. As in earlier sections, e parameterizes the 
signal strength, and we will be using a weak signal ex- 
pansion of expanding in powers o f e a bout e = 0. 

By inse rting the d istrib utions (4.3) and (4.4) into the 
formulae (fTy) and fl2.14|) we obtain 



p(x|e) = exp 



- T o(e,x) 



(4.5) 



where 



5(6, x) 



21n(27r)+ln [1 + 2e] 
(1 + e)al + (1 + e)a 2 
l + 2e 



2ee 



(4.6) 



,. 9 

^2 



3 



3 



and 



(4.7) 



(4.8) 



(4.9) 



The statistic e is the standard cross-correlation statistic. 

Before proceeding further we discuss the validity of the 
weak signal approximation in the context of a stochas- 
tic background signal. Suppose that a stochastic back- 
ground is present and just barely detectable by cross- 
correlating between the two detectors. Then we have 
e ~ 1/ \/N. In the context of ground based detectors like 
LIGO, when this analysis is generalized to colored noise, 
N is replaced by the product TAf, where T is the ob- 
servation time and A/ is the effective bandwidth in the 
usual formula for signal-to-noise ratio [Eq. (1.2) of Ref. 
PI]. Using the estimates T ~ 1/3 year and A/ ~ 50 Hz 
we find 



1 



KT 



(4.10) 



Therefore in all our analyses it will be sufficient to work 
to first order in e. The approximation would only break 
down if e ~ 1, that is, if the stochastic background could 
be seen in a single detector, which is thought to be very 
unlikely. 

We define the statistics d\ and <i 2 by 



di(x) = <ti(x) 2 - 1 - e(x) 



(4.3) and 



d 2 (x)=<7 2 (x) 2 -l-e(x). 



(4.11) 



(4.12) 



Now (a 2 ) = 1 
small; 



e, so the quantities d\ and d 2 will be 



(4.13) 



|di, 2 (x)| <0(l/VN) + 0(e) 



W e now insert Eqs. (4.11) and (4.12) into the formula 
(4.6) for the function S. We expand to second order in 
e, e, d\, and c? 2 , treating these quantities as formally all 
of th e same order . W e then insert the result into Eqs. 
(U|), (|2TTo| ), and (gj), which yields 



A(x) = J depW (e) exp [-N(e - hf + Ne 2 b ] 



(4.14) 



where 



e&(x) = e(x) + - rfi(x)+d 2 (x) 
= ie(x) + i[a!(x) 2 -l] 

+\ h(x) 2 - 1] • 



(4.15) 
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Evaluating the integral over e using the same types of 
arguments as in previous sections gives, using p'°) (e) = 
for e < 0, 



A(x)«e(e 6 )J^p(°)(e 6 )e X p[iVet] 



(4.16) 



Here is the step function. If follows by the same type 
of arguments as before that A(x) ~ O(eb) exp[ATej]. 

Our final result identifies the statistic q, as the opti- 
mal detection statistic; see Ref. [ p| for a more general 
version of this statistic. From Eq. ( [4.15 ) this statistic is 
not the standard cross-correlation statistic e, but instead 
contains the auto-correlation terms a\ — 1 and of — 1. 
The interpretation of these terms is that it is possible, 
under the assumptions of this subsection, to measure the 
stochastic background signal with just one detector. If 
the detector's noise variance is known, then one can just 
measure the variance of the detector's output and sub- 
tract the known noise variance to reveal the stochastic 
background contribution. 

Of course, in reality, the noise in detectors is not known 
priori, and is measured from the data. In particular, 
there is no way that the detectors noise can be known 
beforehand to a fractional accuracy of 10 -4 . Therefore 
we have to generalize the preceding analysis by allowing 
the noise variances to be unknown parameters. 



B. Two coincident co-aligned detectors, white 
Gaussian noise, unknown variances 

We assume that the noise in each detector is white and 
Gaussian with variances a\ and a 2 . We replace Eq. ( |4.3| ) 
with 



Pm(niki) = [[ J- exp 



n i 
'la\ 



(4.17) 



with a similar equation for the second detector. The prior 
distribution for the parameters a\, a 2 will be written as 
p a {o~ii a 2 ). By inse rting the di strib utions (4.17) and (4.4) 
into the formulae (2.11) and (2.14) we obtain 



/>oo />oc 

p(x\e) = / da 1 I da 2 p cr (a 1 ,a 2 )exp 
Jo Jo 



where the function S is now given by 



N . 



-S(e,<7i,o- 2 ) 
(4.18)" 



S(e, 0-1,0-2) = 2 ln(27r) + In [a\a\ + e(af + of)] 
(aj + e)of + (of + t)a\ - 2ee 



+ - 



of of + e(of + of) 



(4.19) 



To evaluate the integral over a\ and 02 in Eq. (4.18), 
we make a change of variables to variables f\ , f 2 defined 
by 



Jl (J l 



(4.20) 



and 



~2 , c t 1 ~ 2 



We also define the rcscalcd variables 



a 



01O2 



a 



O1CT2 



(4.21) 



(4.22) 



We expand 5 to second order around its local minimum 
at /1 = / 2 = 1, a = a: 



2 ln(27r<5-i(72) + 2 + ln(l - a 2 



l + a z 
(1-a 2 ) 2 
2 

(1- A 2 ) 2 



Aa 2 - 



4d 



(1-a 2 ) 2 



Aa(A/i + A/ 2 ) 



(A/i + A/| + 2d 2 Ajf A/f ). (4.23) 



Here A/i = f x - 1, A/ 2 = f 2 - 1, and Aa = a - a. At 
fixed a, S is minimized at 



h = h = 1 



(l + d 2 ) 



(a — a). 



(4.24) 



We now perform the Gaussian integral over a\ , 02 or /1 , 
f 2 , which gives 



p(x|e) 



p^ib(e),a 2b (e)] (1 - d 2 ) 3 / 2 
(27roi02) Ar - 1 4(l + d 2 ) 1 /2^ W 



x exp 



N 



■~.Y2 



2(1 + d 2 



-(a — d) 



(4.25) 



Here <xib(e) is the value of a\ at the peak ( 4.24 ) of the 
integrand, given from Eqs. (4.20) and (4.24) by 



1 



a(a — d) 



1 + d 2 



5?. 



(4.26) 



and similarly for a 2 b (e)- The factor J7(e) is a Jacobian 
factor given by 



J{e) = 1 - 



/i 2 o-i 



-1/2 



1 - 



/2^2 2 



-1/2 



(4.27) 



wher e / 1 and f 2 are given in terms of e by Eqs. (4.22) 
and (pj~24l ) 



We nex t inse rt the result (4.25) for p(x|e) into Eqs. 
and ( |2.10D . The result is 



A(x) = / dep^je ff^Y^l ^A 
1 P K Pa[a U (0),a 2b (0)}J(0) 



x exp 



N 



2(1 + d 2 ) 
Finally, integrating over e gives 



(a 2 — 2ad) 



(4.28) 



A(x) 



2?r(l + d 2 ) ( o )( ^ p CT [<7ib(e),o-2fc(e)] >7(e) 
iV P [£> pa[a lb (0),a 2b (0)]j{0) 



x exp 



7Vd 2 " 
2(1 + d 2 ). 



9(d), 



(4.29) 
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and invoking the arguments of the Sees. Ill A and [II B 
above for neglecting the prefactors gives 



A(x) ~ exp 



Na 2 



2(1 



9(d). 



(4.30) 



Thus, in the limit d< 1, A(x) is equivalent to the usua l 
cross-correlation statistic 0(&)& defined by Eqs. (4.7), 
@, Q, and ( ggg ). 

We end this subsection by recapitulating the various 
approximations necessary to obtain the result: 



• The large N approximation N 3> 1, necessary for 
the validity of the Laplace approximation in inte- 
grating over o i , (72. 

• The assumption that we are in the regime where 
the signal is detectable, exp[A^d 2 /2] ^> 1. This is 
necessar y for the evaluation of the integral over e 
in Eq. (4.28) , and for neglecting the prefactors in 
deriving Eq. (4.3C). 



• The assumption, as before, that the prior probabil- 
ity distributions p(°\e) and p a {p\,<Ji) ar e slowly 
varying. 

• The weak signal approximation a <C 1, which will 
be satisfied unless the stochastic background con- 
tribution to the output of one of the detectors be- 
comes comparable to the noise in that detector. As 
discussed above, for signal strengths at the margin 
of dctectability, and for several month searches for 
a stochastic background with ground based inter- 
ferometers, we have a ~ 10 -4 . 



Two coincident, co-aligned detectors, white 
non-Gaussian noise 



We next turn to the non-Gaussian noise model of Sec. 
III. A. of paper I. The noise in each detector is assumed 
to be white, with each sample statistically independent 
and identically distributed, so that 



p n (ni,n 2 ) = JJexp[-/i(nij) - fa{n 2 j)] 



(4.31) 



However, it is clear that we cannot assume that the noise 
distributions e - ^ 1 and e~^ 2 in each detector are known in 



advance. Otherwise, as explained in Sec. IV A, the anal- 
ysis would predict that one can measure the stochastic 
background in a single detector by measuring the noise 
distribution and subtracting from it the "known" noise 
distribution. 

Therefore, in this subsection, we will allow the func- 
tions fi(x) and fa(x) to be arbitrary except for the nor- 
malization conditions 



e- fl{x) dx = 



~ f2 ^dx = 1. 



(4.32) 



Formally, there are an infinite number of parameters to 
specify to determine the functions fa and fa. However, 
in practice these distributions will be measured as his- 
tograms, determined by a finite set of numbers or pa- 
rameters. We identify this finite set o f parameters with 
t he n oise parameters 6 n of Eq. ( 2.14 ). We rewrite Eq. 
( ETUI) as 



p n (m,n 2 ) = J Vfa J Vfap n (n 1 ,n 2 \fa,fa)pf[fa,fa}- 

(4.33) 

Here for simplicity we have used a functional or path in- 
tegral notation for the integral over the noise parameters 
(even though the inte gral is only over a finite number of 
parameters). In Eq. ( 1.33] ), Pf[fa, fa) is the prior proba- 
bility density functional for the functions fa and fa, and 
p n (ni, 112I/1, fa) is given by the expression on the right 
hand side of Eq. ( 4.31 ). The probability distribution 
Pf [fa 1 fa] encodes our assumption that the noise distribu- 
tions will have central Gaussian regions, with unknown 
variances o\ and o 2 , and arbitrary tail regions containing 
a small fraction p t aii of the total pr obab ility. 

We now insert the distributions (4.4) and ( |4.33j ) into 
Eqs. (2.11) and (2.14) and expand to second order in e. 
The result is 



p(x|e) = J VfaVfa Pf [faJ 2 }Y[e-M x 



Y[ [1 + eAj + eCj + e 2 £ 3 + 0(e 3 )] ,(4.34) 



where 



and 



C 3 = fl( X lj)f2( x 2j)- 



(4.35) 



(4.36) 



The quantity £j is a sum of terms of the form 

f[ n \ Xlj ) m , jfW m , and f[ n \x }j rti l \x 2] y for in- 
tegers n, to, I, r, whose exact form will not be needed here. 

Working to second order in e, we can re-express Eq. 
(pi as 



p(x|e) = J Vfa J Vfap f [fa,fa] 

xexp[-S(e,/ 1 ,/ 2 )], (4.37) 



where 



E(e,fa,fa) = S [/i,/ 2 ]+eS 1 [/ 1 ,/ 2 ]+e 2 S 2 [/ 1 ,/ 2 ] 

+0(e 3 ), (4.38) 



where 



S [/i,/ 2 ] = -Y,fi(x lj ) + f2(x2 j ), (4.39) 
3 
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Si[/i,/ 2 ] 



3 



(Aj + Cj), 



and 



3 2 [/i,/ 2 ]=£ 



(4.40) 



(4.41) 



We now note that we c an e liminate the auto- 
correlation terms Aj from Eq. ( 1.4C| ) by making a change 
of variables. We define the operator V e that acts on func- 
tions via 



(V e f)(x)=f(x)-±ef(x) 2 + ±ef"(x), 

and we define the functions F\ and F% by 
F 1 =VeA, F 2 =VJ 2 . 



(4.42) 



(4.43) 



Using Eqs. ( pL35| ), (pU^-(pO[), (g^42J) and ( pL43[ ) the 
functional 3 can be rewritten as 

H(e,/i,/ 2 ) = So^^^J+eSi^^^l+e 2 ^^,^] 
+(9(e 3 ). (4.44) 

Here the first order piece Si consists only of the cross- 
correlation term, 

Silfi.Ja] =J2 F i( x ij) F 2^2j); (4.45) 

i 

the corrections to this cross-correlation expression due 
to changing from /i, / 2 to Fi, F 2 do not appear at this 
(linear) order in e and instead contribute to the second 
order term S 2 [Fi , F 2 ] . The exact form of the functional 
3 2 [iq, F 2 ] will not be needed for our arguments below. 

We define the functions A (x) and f 2 (x) to be the func- 
tions corresponding to the measured noise distributions 
at the two detectors. That is, they are step functions 
defined by the requirement 



- E 

j with xi j <x 



1. 



(4.46) 



with a similar equation for f 2 . 

Consider now the evalu ation of the integral (4.37) with 
3 given by the expression ( 4.44 ). Consider first the e — > 
limit. In this limit one can show from the normalization 



conditions ( 4.32 ) that 3 will be minimized at the mea- 
sured noise distributions: 



Fi—fl, F 2 — f 2 



(4.47) 



For nonzero e, the leading order correction to t he e = 
result will be given by evaluating the function (4.44) at 
the local minimum (4.47). We thus arrive at 



p(x|e) = J^)pf[V7 X kV7 X U exp{-Ho[/i,/a]} 

exp {-eSilA, f 2 ] - e 2 3 2 [A, A]} ■ (4-48) 



Here »/(x) is a width factor whose origin is approximat- 
ing the integrals over fx and f 2 as the value of the in- 
tegrand at the peak times the "width" of the peak [ p^[ . 
This factor is analogous to the variou s fact ors that ap- 
pear in front of the exponential in Eq. (4.25). It depends 
weakly on x in comparison to the exponential factors. 

The second order functional S 2 in Eq. ( [1.48 ) will differ 
from the corresponding functional S 2 in Eq. ( 4.44 ) since 
the location of the peak of the integrand will re ceive a 
correction of order e away from the value (4.47) which 
will give a correction of order 0(e 2 ) to the value of the 
integral. 



W e now inser t the formula (4.4S) for p(x, e) into Eqs. 
(U) and ( )2.10D . This gives 



A(x) = / W°>(e) ^^^ 



Pf[h,h] 

x expj-eHitA,/,] - e 2 l 2 [A, f 2 }} .(4.49) 
Evaluating the integral over e gives 




x exp 



9(30, (4.50) 



.43 2 [/!,/ 2 ] 

and as before we can neglect the prefactors to give 



A(x) 



exp 



Si[/i,/ 2 ] 2 

.4i 2 [A,/ 2 



9(30. 



(4.51) 



Now the statistic 3i[A>/ 2 ] defined by Eqs. (4.45) and 
( 4.46 ) coincides with the locally optimal statistic ob- 
tained in paper I [Eq. (3.8) of paper I, specialized to 
a white stochastic background], except for the following 
modification. One first measures the noise prob abili ty 
distributions in each detector separately [cf. Eq. ( 4.46 )1- 
Then, o ne co mputes the generalized cross-correlation 
statistic (4.45) using the those distributions and the mea- 
sured data. 

From Eq. (4.51), we see that A(x) will be ap- 
proximately equivalent to the locally optimal statistic 

Si [A 7 A] if the statistic 3 2 [A,A] has a weak depen- 
dence on the data x. To establish this, consider the limit 
Ptaii — * 0, where ptail "C 1 is the total probability in the 
noise distribution tails. In that limit our assumptions im- 
ply that the noise distributions in the two detectors are 
Gaussians with unknown variances a% , tr 2 , and therefore 
the analy sis of this subsection reduces to the analysis of 
Sec. [VB| above. Therefore we can read off the p t ail - * 



limit of th e statistic 3 2 [/i,/ 2 ] by comparing Eqs. ( 4.28| ) 
and ( 1.49| ) and identifying the coefficients of e 2 in the 
arguments of the exponentials. We thus obtain 



s 2 [A,A] = 



N 



2(1 



i-2 i.2 



[1 + OOtaii)] ■ (4.52) 



12 



In th e lim it a <C 1 we can neglect the a dependence in 
Eq. ( 4.52 ). We then see that the x-dependent fluctua- 
tions in the statistic are sup presse d b y eith er the small 
parameter ptaii, as as in Sees. [II C and III D above, or by 
the parameter 1 / yN governing the size of the fractional 
fluctuations of the statistics ai, &2- Thus, we can neglect 



the x dependence of ^[/l, j^] in Eq. (4.51), and the ap- 
proximate equivalence of A(x) and the locally optimal 
statistic Si[ji,/ 2 ] follows. 

Finally, we remark that we have not analyzed, in this 
paper, the most general situation for stochastic signals 
of separated, non-aligned detectors with colored noise, 
which was analyzed in Sec. IV. B of paper I How- 
ever, the results we have obtained make it very plausible 
that, for that more general situation, the Bayesian statis- 
tic A(x) should again be equivalent to the generalized 
cross-correlation statistic derived in paper I. 

We also note that our assumption that the stochastic 
signal be Gaussian is necessary for our analysis. Mod- 
ifying the signal by making it be non-Gaussian instead 
of Gaussian would alter Eqs. ([O]), ( [4.19[ ) and ( f4.38| ) at 
0(e 2 ). Therefore, the derivation here does not generalize 
straightforwardly to non-Gaussian stochastic signals, un- 
like the corresponding derivations in paper I. In Rcf. |^| 
it is shown that one can find detection techniques tailored 
to non-Gaussian stochastic signals that perform better, 
for such signals, than the methods considered here. 



V. CONCLUSION 

The derivation in this paper, from a different frame- 
work, of the detection strategies obtained in paper I gives 



us increased confidence in the utility of those strategies. 
In addition, the analysis of this paper has clarified the 
regime in which we expect the strategies to work well. 
For deterministic signals, data segments to be analyzed 
should be long enough that the signal-to-noise squared 
per data point be small. This requirement is easy to sat- 
isfy in practice, as signal-to-noise thresholds are usually 
in the range 5 — 10. In addition, the strategies will only 
be close to optimal in the regime where signals are strong 
enough to be detectable; this restriction is unimportant 
in practice, as the performance of detection statistics in 
the regime where signals are far too weak to be detected is 
not important. Finally, for stochastic signals, the signal 
must be small compared to the noise in each individual 
detector, and the total probability in the tail part of the 
noise distributions must be small. 
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